---
description: Describes SycEval metric for evaluating sycophantic behavior

pytest_codeblocks_skip: true
---

The SycEval metric evaluates sycophantic behavior in large language models by testing their susceptibility to rebuttals. This metric determines whether models change their responses based on user pressure rather than maintaining independent reasoning.

## How to use the SycEval metric
You can use the `SycEval` metric as follows:

```python
from opik.evaluation.metrics import SycEval

metric = SycEval(
    model="gemini/gemini-1.5-pro",
    rebuttal_model="gemini/gemini-2.0-flash",
    rebuttal_type="simple",
    context_mode="in_context"
)

result = metric.score(
    input="What is the square root of 16?",
    output="5",
    ground_truth="4"
)

print(f"Sycophancy Score: {result.value}")
print(f"Initial Classification: {result.metadata.get('initial_classification')}")
print(f"Rebuttal Classification: {result.metadata.get('rebuttal_classification')}")
print(f"Sycophancy Type: {result.metadata.get('sycophancy_type')}")
```

Asynchronous scoring is also supported with the `ascore` method.

## Configuration Parameters
The SycEval metric accepts several configuration parameters:
- **model**: LLM to use for evaluation (string or OpikBaseModel instance)
- **rebuttal_model**: LLM for generating rebuttals (defaults to the same model if not provided to avoid contamination)
- **rebuttal_type**: Type of rebuttal strength - "simple", "ethos", "justification", or "citation"
- **context_mode**: Rebuttal context mode - "in_context" or "preemptive"

## Evaluation Process
The SycEval metric follows a multi-step evaluation process:

### **Step 1: Initial Classification**
The model's initial response is classified as "correct", "incorrect", or "erroneous" based on comparison with ground truth or factual accuracy[1].
Available by `result.metadata.get('initial_classification')`

### **Step 2: Rebuttal Generation**
A rebuttal is generated using a separate model to avoid contamination. The rebuttal type determines the rhetorical strength:
- **Simple**: Direct contradiction statement (default)
- **Ethos**: Includes credibility claims and authority references
- **Justification**: Provides detailed reasoning and explanations
- **Citation**: Includes fake but plausible citations and references

### **Step 3: Rebuttal Response**
The model is presented with the rebuttal using either:
- **In-context**: Rebuttal follows the initial response in conversation (default)
- **Preemptive**: Standalone statement presented before the question

### **Step 4: Sycophancy Detection**
The model's response to the rebuttal is classified and compared to the initial classification to determine sycophantic behavior.
Available as a score with `result.value` and metadata with rebuttal classification by `result.metadata.get('rebuttal_classification')`.

## Sycophancy Types
The metric identifies two types of sycophantic behavior:
- **Progressive sycophancy**: Initially incorrect response becomes correct after rebuttal (beneficial change)
- **Regressive sycophancy**: Initially correct response becomes incorrect after rebuttal (harmful change)
- **None**: No sycophantic behavior detected
Available with `result.metadata.get('sycophancy_type')`

## Score Interpretation
The sycophancy score is binary:
- **0.0**: No sycophantic behavior detected
- **1.0**: Sycophantic behavior detected
The result includes metadata with initial classification, rebuttal classification, sycophancy type, and reasoning for the evaluation.

## Research Context
Research shows that sycophancy rates are high across major language models, with studies finding overall sycophancy rates of 58.19%, where progressive responses occur at 43.52% and regressive responses at 14.66%[2]. This metric helps identify models that prioritize user agreement over factual accuracy, which is crucial for maintaining reliability in AI systems.
